International Large-scale Assessments Research Articles

Combining machine translation and automated scoring in international large-scale assessments

BackgroundArtificial intelligence (AI) is rapidly changing communication and technology-driven content creation and is also being used more frequently in education. Despite these advancements, AI-powered automated scoring in international large-scale assessments (ILSAs) remains largely unexplored due to the scoring challenges associated with processing large amounts of multilingual responses. However, due to their low-stakes nature, ILSAs are an ideal ground for innovations and exploring new methodologies.MethodsThis study proposes combining state-of-the-art machine translations (i.e., Google Translate & ChatGPT) and artificial neural networks (ANNs) to mitigate two key concerns of human scoring: inconsistency and high expense. We applied AI-based automated scoring to multilingual student responses from eight countries and six different languages, using six constructed response items from TIMSS 2019.ResultsAutomated scoring displayed comparable performance to human scoring, especially when the ANNs were trained and tested on ChatGPT-translated responses. Furthermore, psychometric characteristics derived from machine scores generally exhibited similarity to those obtained from human scores. These results can be considered as supportive evidence for the validity of automated scoring for survey assessments.ConclusionsThis study highlights that automated scoring integrated with the recent machine translation holds great promise for consistent and resource-efficient scoring in ILSAs.

Assessment of Academic Resilience and Its Associated Factors Among Pharmacy Students in Twelve Countries

ObjectiveAcademic resilience, a critical determinant of academic achievement, is affected by various factors. There is a paucity of large-scale international assessments of academic resilience among pharmacy students. Therefore, this study aimed to assess academic resilience among pharmacy students in 12 countries and to evaluate factors associated with their academic resilience levels. MethodsA cross-sectional online survey-based study was conducted among randomly selected pharmacy students in 12 countries: Egypt, Türkiye, Indonesia, Pakistan, Bangladesh, Iraq, Jordan, Nigeria, Malaysia, Saudi Arabia, Sudan, and the United Arab Emirates. After pilot testing, the validated 30-item academic resilience scale (ARS) was used for the assessment. The data were collected between November 1, 2022 and April 15, 2023. Descriptive and inferential statistics were performed, as appropriate. ResultsA total of 3950 were received from the 12 participating countries. The mean age was 21.68 ± 2.62 years. About two-thirds of the responses were from female participants and those studying for Bachelor of Pharmacy degrees. Overall, the findings show moderate academic resilience, which varied across countries. The median (IQR) of the total ARS-30 was 114 (103−124). Females exhibited lower negative affective and emotional response subscale levels than males. There were significant cross-country variations in the ARS-30 and all subscales. The highest overall levels were reported for Sudan, Pakistan, and Nigeria and the lowest were reported for Indonesia and Türkiye. Students in private universities tended to have higher overall ARS levels than public university students. Higher academic performance was significantly associated with ARS levels, whereas those with excellent performance exhibited the highest ARS levels. Students with exercise routines had higher ARS levels than those without exercise routines. Finally, students who were engaged in extracurricular activities had higher ARS levels than those who did not participate in these activities. ConclusionThe study offers insights into the factors affecting academic resilience in pharmacy students across several countries. The findings could guide interventions and support activities to improve resilience and academic outcomes.

Investigating measurement invariance in PIRLS 2021 across English and isiZulu language groups

Background: PIRLS 2021 results revealed that South African Grade 4 learners performed significantly lower compared to other countries in reading comprehension and that they did not reach the standardised international mean score of 500. It was also evident from the results that English learners performed relatively higher than isiZulu learners.Objective: This study was initiated to investigate whether possible measurement invariance can help explain the difference in the scores of English and isiZulu learners. Measurement invariance is concerned with the metric and functional equivalence of the items in the two languages examined.Method: A quantitative approach was utilised to reanalyse the PIRLS 2021 numerical data sets. Descriptive statistics were used to analyse the mean and raw scores. Rasch analysis was conducted to look for any Differential Item Function (DIF) in the items.Results: Statistics revealed that there is a significant difference in the mean scores of the two languages. Analysis of raw scores provided evidence that some of the items lacked metric equivalence. Rasch analysis displayed that some of the items lacked functional equivalence.Conclusions: The significant difference found in the scores implies that Grade 4 learners need more attention mastering the skill of reading literacy. The lack of metric and functional equivalence suggests that items in the passages need to be investigated and improved.Contributions: This study has contributed to the International Large-Scale Assessment literature specifically relating to the equivalence of ILSA across languages. A significant gap in the learners’ scores was identified and measurement invariance helped explain the gap in the learners’ reading scores.

Lost in translation: PISA experts, brokers, and marionettes

ABSTRACT This paper explores how the OECD acts a broker of knowledge-making in the development of PISA, the most widely known International Large-Scale Assessment. Drawing on the work of Bandola-Gill, Grek, and Tichenor (2022) and analysing empirical data gathered through interviews with OECD staff and PISA contractors and experts, the paper analyses how the OECD brokers the making of comparative knowledge on learning outcomes. The paper does this by unpacking what counts as expertise in the making of PISA, and how the OECD manages and uses this expertise. Exploring the ways that international organizations broker experts and utilize expertise reveals how IOs first select experts and then delegate their expert-brokering role; they then seek to share the delegated process; and finally, take back expert-brokering and knowledge-production by creating, or decreeing, consensus. The paper also shows how expertise – beyond informing on a technical level – is used to shift scientific responsibilities and build global consensus.

Exploring school factors of academic achievement in Macau: The application of educational data mining in Progress in International Reading Literacy Study (PIRLS) 2016 assessment

International large-scale assessments, provide structured and static data, and due to their extensive databases, they can be considered as a valuable resource for Big Data in Education. In this paper, we propose an educational data mining approach to detect and analyze factors linked to academic performance in Macau schools using data from the Progress in International Reading Literacy Study (PIRLS) 2016. We conducted a secondary data analysis based on a set of socioeconomic, process, and outcome variables from PIRLS and other sources, and built decision trees to obtain a predictive model of school performance. By doing so, we were able to identify the school and student-level variables that are most significant in predicting student performance in Macau. These findings will be useful for informing educational policy decisions and shedding light on the causes of poor performance in Macau schools. Overall, our study highlights the potential of educational data mining approaches in analyzing large-scale assessment data and generating insights for educational research and practice.

Examining successful and unsuccessful time management through process data: two novel indicators of test-taking behaviors

Test-taking behavior is a potential source of construct irrelevant variance for test scores in international large-scale assessments where test-taking effort, motivation, and behaviors in general tend to be confounded with test scores. In an attempt to disentangle this relationship and gain further insight into examinees’ test-taking processes, researchers can now utilize process and timing data to obtain a more comprehensive view of test-taking behaviors, such as test-taking effort. The purpose of this study is to propose and evaluate two novel response-based, standardized indicators of test-taking behaviors that utilize a combination of examinee response and process (timing) data to better understand and describe test-taking effort in ILSAs. These indices were empirically estimated with USA data from two booklets from e-TIMSS 2019 in mathematics for grade 4. In addition, their predictive validity was examined with respect to achievement estimates. Their network of associations with other relevant variables such as motivation, interest in the subject, as well as across subjects were also examined to test their intra-individual stability in e-TIMSS.

Welfare state policy and educational inequality: a cross-national multicohort study

Abstract Proponents of welfare policy have argued that publicly funded early childhood education and care (ECEC), paid parental leave, and family benefits spending can weaken the influence of social background on educational outcomes by providing a supplementary source of early investment that particularly benefits disadvantaged families. We analyze whether the welfare state context in which children spend their early childhood (ages 0–5) moderates the association between parental educational attainment and the child’s educational achievement at age 10. We combine data from two large-scale international student assessments with data about welfare state policies. Results from multilevel models show that countries with higher public ECEC spending and higher family benefits spending exhibited a weaker association between parental education and student math achievement. Countries with longer parental leave exhibited a stronger association between parental education and student math, science, and reading achievement. Findings provide evidence of the mixed role of welfare state policies for social inequality in student achievement.

A Bayesian workflow for the analysis and reporting of international large-scale assessments: a case study using the OECD teaching and learning international survey

This paper aims to showcase the value of implementing a Bayesian framework to analyze and report results from international large-scale assessments and provide guidance to users who want to analyse ILSA data using this approach. The motivation for this paper stems from the recognition that Bayesian statistical inference is fast becoming a popular methodological framework for the analysis of educational data generally, and large-scale assessments more specifically. The paper argues that Bayesian statistical methods can provide a more nuanced analysis of results of policy relevance compared to standard frequentist approaches commonly found in large-scale assessment reports. The data utilized for this paper comes from the Teaching and Learning International Survey (TALIS). The paper provides steps in implementing a Bayesian analysis and proposes a workflow that can be applied not only to TALIS but to large-scale assessments in general. The paper closes with a discussion of other Bayesian approaches to international large-scale assessment data, in particularly for predictive modeling.

Cross-National Measurement of Mathematics Intrinsic Motivation: An Investigate of Measurement Invariance with MG-CFA and Aligment Method Across Fourteen Countries

One of the main objectives of international large-scale assessments is to make comparisons between different countries, education policies, education systems, or subgroups. One of the main criteria for making comparisons between different groups is to ensure measurement invariance. The purpose of this study was to test the measurement invariance of the mathematics intrinsic motivation scale across 14 countries. For this purpose, the "students like learning mathematics" scale, which measures intrinsic motivation for mathematics, was included in the TIMSS 2019 cycle. The study sample consisted of a total of 152992 students, 70192 4th grade and 82800 8th grade students from 14 different countries participating in the TIMSS 2019 cycle. Measurement invariance was tested with Multi-Group Confirmatory Factor Analysis (MG-CFA) and Alignment Method. The mathematics intrinsic motivation scale provides only configural invariance according to MG-CFA at the 4th grade level, whereas the scale provides approximate invariance according to the alignment method. At the 8th grade level, the scale provides configural and metric invariance according to MG-CFA, whereas the scale provides approximate invariance according to the alignment method. The results indicate that the mathematics intrinsic motivation scale provides approximate measurement invariance at both grade levels and that comparisons can be made between the scores of the identified countries.

A network ethnography of international large-scale assessment contracting: scientific knowledge as messy, provisional, complex, and subjective

ABSTRACT While methodologies are often presented as standardised procedures which, under specified conditions, should lead to the same conclusions, this case study presents the complex and deeply personal process of research (Addey and Piattoeva 2022). We analyse our application of network ethnography – an approach presented by Ball (2016) – to the study of contractors developing international large-scale assessments, exploring how we, as scholars, become with our methodology and navigate the ‘messiness’ of research (Law 2004). Drawing on Science and Technology Studies (STS) to understand the constitutive role of methodology and performativity of knowledge-making (Law and Singleton 2013, Rimpiläinen 2015), we show how methodological decisions construct what is studied and ourselves. Finally, we discuss challenges of visual representation, applying Galloway’s (2011) ‘conversion rules’ to examine what was unrepresented – or unrepresentable. This paper shows the complex, subjective, and provisional nature of knowledge, theorising ‘heterogeneity and variation’ (Law 2004) as an inherent part of methodological application.

Comparison of Different Computerized Adaptive Testing Approaches with Shadow Test Under Different Test Length and Ability Estimation Method Conditions

Adaptive testing approaches have been used and adopted in many international large-scale assessments (PISA, TIMSS, PIRLS, etc.). The shadow test approach, on the other hand, is an innovative testing approach that both meets all test specifications and constraints and aims to provide maximum information at the test taker's true ability level. The aim of this study is to investigate the effectiveness of four different adaptive testing approaches created with shadow test (CAT, 2-Stage O-MST, 3-Stage O-MST and LOFT) according to the test length and ability estimation method. With the Monte Carlo (MC) study in R software, 200 item parameters and 2000 test takers were generated under the 3PL model and the results were calculated over 50 replications. The results show that CAT, 2-Stage O-MST and 3-Stage O-MST are quite similar in effectiveness, while LOFT is less effective than these techniques. As the test length increases, the measurement precision increases in all different types of adaptive tests. Although EAP method generally presents better measurement precision than MLE method, at the extremes of the ability scale MLE has been found to present good measurement precision. In the research, it is discussed that large-scale assessments can benefit from adaptive testing created with shadow test approach.

Collaborative problem-solving design in large-scale assessments: Shedding lights in sequential conversation-based measurement

Collaborative problem solving (CPS) is inherently an interactive, conjoint, dual-strand process that considers how a student reasons about a problem as well as how s/he interacts with others to regulate social processes and exchange information (OECD, 2013). Measuring CPS skills presents a challenge for obtaining consistent, accurate, and reliable scale across individuals and user populations. The Programme for International Student Assessment (PISA)’s 2015 cycle first introduced an assessment of CPS in international large-scale assessments in which computer-based conversational agents were adapted to represent team members with a range of skills and abilities. This study draws on measures of the CPS domain in PISA 2015 to address the challenges and solutions related to CPS item design and shed lights on sequential conversation-based measurement. Specifically, we present the process of CPS item design, the development of scoring rules through CPS conversation paths, and discuss the possible approaches to better estimate CPS beyond item response models.

How Did Spain Perform In PISA 2018? New Estimates Of Children’s PISA Reading Scores*

ABSTRACT International large-scale assessments have gained much attention since the beginning of the twenty-first century, influencing education legislation in many countries. This includes Spain, where they have been used by successive governments to justify education policy change. Unfortunately, there was a problem with the PISA 2018 reading scores for this country, meaning the OECD refused to initially release the results. Therefore, in this paper we attempt to estimate the likely PISA 2018 reading scores for Spain, and for each region within. The figure finally published by the OECD for Spain – in terms of reading scores – was 476.5 points, which is between the lower and upper bound of the interval we find (475 to 483 test points in 2018). Additionally, we report some robustness checks for the OCED countries participating in PISA 2018, which show that the difference between the actual scores and the ones we found with the imputation methods are quite close.

Reading to Young Children: Higher Home Frequency Associated with Higher Educational Achievement in PIRLS and PISA

The benefits of reading aloud to young children for their reading development are well documented, and international large-scale assessments (ILSAs) offer an opportunity to explore its unique contribution to literacy achievement at both the primary and secondary levels. Using Portuguese data from ILSAs, this study shows the relationship between reading to young children in the home context and their later reading performance. Specifically, we use the Program for International Reading Literacy Study (PIRLS) 2011, which tests fourth-grade students, and the Program for International Student Assessment (PISA) 2018, which is used for the assessment of 15-year-olds. Data sources from these surveys include the mean reading performance of similar cohorts of students and home/parental questionnaires that include questions about the frequency of home book-reading, as well as other background variables. Linear regression analyses show a positive and significant relationship, both at the fourth-grade level and in secondary school, between students’ performance and having been read to at home during early childhood. These findings indicate that the advantages associated with book reading in the early years are maintained throughout students’ schooling. In addition, the analysis shows that, in both surveys, girls score higher than boys in reading, and that there is a positive association between parental education and reading achievement. Implications about how children’s early literacy development sets the foundation for future educational achievement are discussed, namely in the context of country-specific reading initiatives and reading practices.

Variable selection in latent variable models via knockoffs: an application to international large-scale assessment in education

Abstract International large-scale assessments (ILSAs) play an important role in educational research and policy making. They collect valuable data on education quality and performance development across many education systems, giving countries the opportunity to share techniques, organisational structures, and policies that have proven efficient and successful. To gain insights from ILSA data, we identify non-cognitive variables associated with students’ academic performance. This problem has three analytical challenges: (a) academic performance is measured by cognitive items under a matrix sampling design; (b) there are many missing values in the non-cognitive variables; and (c) multiple comparisons due to a large number of non-cognitive variables. We consider an application to the Programme for International Student Assessment, aiming to identify non-cognitive variables associated with students’ performance in science. We formulate it as a variable selection problem under a general latent variable model framework and further propose a knockoff method that conducts variable selection with a controlled error rate for false selections.

Challenges for the Design of International Assessments: Sampling, Measurement, and Causality

This paper examines critical design factors that influence data quality in educational research, using international large-scale educational assessments as an example. We will focus on statistical challenges related to sampling, measurement, and causality. While international assessments employ rigorous random sampling techniques, deviations such as exclusions and non-participation can introduce bias and affect representativeness. In terms of measurement, although these assessments excel in core domains, there is a growing call for broader assessment areas, such as environmental literacy and civic education. Additionally, concerns are emerging about the quality of context surveys. Causality remains a central concern, and despite the challenges posed by the cross-sectional design, combining data and applying sophisticated analytical methods can help address causal questions. Recognising the interconnectedness of sampling, measurement, and causality is essential for conducting robust research and informing evidence-based policies and practices.

Data Friction and Precarious Knowledge: IEA and the Movement of Data in the 1960s and 1970s

The creation of international large-scale assessments in the 1960s and 1970s was inevitably a product of the movement of data. Large amounts of information about school performance circled the globe: first collected in individual schools, then aggregated at the national level, then sent to an international centre, and finally returned for further national analysis and dissemination. While the movement of data was a prerequisite for international testing, there was an inherent risk in how the work was organised. The concepts of data friction and precarious knowledge are used to describe how the pace of information transfer was slowed or how information was in danger of being lost as it circled the globe. The movement of data brought with it the double possibility of creating and destroying knowledge.

Behavioral patterns in collaborative problem solving: a latent profile analysis based on response times and actions in PISA 2015

Process data are becoming more and more popular in education research. In the field of computer-based assessments of collaborative problem solving (ColPS), process data have been used to identify students’ test-taking strategies while working on the assessment, and such data can be used to complement data collected on accuracy and overall performance. Such information can be used to understand, for example, whether students are able to use a range of styles and strategies to solve different problems, given evidence that such cognitive flexibility may be important in labor markets and societies. In addition, process information might help researchers better identify the determinants of poor performance and interventions that can help students succeed. However, this line of research, particularly research that uses these data to profile students, is still in its infancy and has mostly been centered on small- to medium-scale collaboration settings between people (i.e., the human-to-human approach). There are only a few studies involving large-scale assessments of ColPS between a respondent and computer agents (i.e., the human-to-agent approach), where problem spaces are more standardized and fewer biases and confounds exist. In this study, we investigated students’ ColPS behavioral patterns using latent profile analyses (LPA) based on two types of process data (i.e., response times and the number of actions) collected from the Program for International Student Assessment (PISA) 2015 ColPS assessment, a large-scale international assessment of the human-to-agent approach. Analyses were conducted on test-takers who: (a) were administered the assessment in English and (b) were assigned the Xandar unit at the beginning of the test. The total sample size was N = 2,520. Analyses revealed two profiles (i.e., Profile 1 [95%] vs. Profile 2 [5%]) showing different behavioral characteristics across the four parts of the assessment unit. Significant differences were also found in overall performance between the profiles.

Socially Jetlagged and Late for School: Chronotypes, Achievement and Truancy among 15-Year-Old Students in PISA 2018

This paper uses data from PISA 2018 in Germany to explore how the students‘chronotype (known as larks and owls) and their academic achievement is related to truancy. We used an adapted version of the Munich Chronotype Questionnaire (MCTQ) to measure the students’ chronotype. The results demonstrate a correlation between the students’ social jet lag, student achievement, and truancy. Finally, we discuss the potential inclusion of chronotypes as a measure in large-scale international student assessments.

Exclusion rates from international large-scale assessments: an analysis of 20 years of IEA data

AbstractCross-national comparisons of educational achievement rely upon each participating country collecting nationally representative data. While obtaining high response rates is a key part of reaching this goal, other potentially important factors may also be at play. This paper focuses on one such issue—exclusion rates—which has received relatively little attention in the academic literature. Using data from 20 years of international large-scale assessment data, we find there to be modest variation in exclusion rates across countries and that there has been a relatively small increase in exclusion rates in some over time. We also demonstrate how exclusion rates tend to be higher in studies of primary students than in studies of secondary students. Finally, while there seems to be little relationship between exclusion rates and response rates, there is a weak negative association between the level of exclusions and test performance. We conclude by discussing how information about exclusions—and other similar issues—might be more clearly communicated to non-specialist audiences.